Skip to content

fix(benchmark): integrate terminal preservation and explicit quota continuation - #8219

Closed
huangruiteng wants to merge 4 commits into
codex/edgebench-integration-r30-20261107from
codex/edgebench-integration-r31-20261107
Closed

huangruiteng wants to merge 4 commits into
codex/edgebench-integration-r30-20261107from
codex/edgebench-integration-r31-20261107

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Failed benchmark workers could lose their latest recoverable filesystem during cleanup, while repaired account quota failures had no explicit same-session continuation in the outer scheduler. This integration combines #8216 and #8217 on r30: opt-in failure preservation stops the worker before grading drain, and digest-bound continuation retains the original failure/session/attempts inside a separately admitted finite phase.

Both component heads received independent exact-head reviews: preservation, continuation. The integration introduces no additional behavior edits; all 21 changed files are byte-identical to their reviewed owning head. Default cleanup and automatic quota retry behavior remain unchanged.

Validation at b0dd6bdb5a0724a74a20f10ded4208f2fbb4a388:

  • 18 production preservation tests and 14 combined boundary tests passed, including real SForge SDK, isolated Docker, native HTTP grading and SQLite/worker subprocess paths.
  • Combined recovery suite: 111 passed / 10 skipped on the host; seven missing-SDK cases passed with the existing Linux provider. Three optional Docker cases reuse the unchanged exact component qualification, rather than claiming a new run.
  • 34 TypeScript effect-runtime checks passed.
  • Actual work-image installation passed with the selected SDK: four generated Native Goal startup/failure scenarios, full heartbeat bootstrap/Explore configuration, installed continuation entrypoint and invalid-digest rejection before any host/phase. The prior bundle was correctly refused after three build inputs changed; the rebuilt bundle and all 717 input bindings were validated. Linux operator-side preservation rejects missing controller configuration before dispatch. Containers were isolated, sequential and cleaned; zero model or scoring calls. An initial synthetic fixture ordering failure is retained separately from the corrected passing attempt.
  • Premerge: six direct and nineteen selected checks passed, zero failures/warnings; overall remains manual_review_required for the benchmark-sensitive boundary.
  • Merge-base, DCO, staged bilingual RFC check, whitespace and public-boundary checks passed. No rendered frontend changes.

Raw failure capture is not a completed research checkpoint. Missing historical workspace, ambiguous native failure, unavailable allowance, changed session/profile or Stop remain held. These tests use isolated deterministic fixtures; they do not demonstrate live historical restoration, scientific score gains or paid inference. Installed qualification covers prerequisites and fail-closed entrypoints, not positive live restoration or model health. Runtime deployment still requires per-run authority/resource checks. No main merge is included.

Main delivery

The two constituent PRs now target main. The original stop-before-grading and explicit quota-continuation changes continue independently through their original PRs. Preservation #8216 · Continuation #8217

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…ed sessions

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max

HEAD: b0dd6bdb5a0724a74a20f10ded4208f2fbb4a388;实际目标为 codex/edgebench-integration-r30-20261107,不可变 base 1a309ccb3278b68f79d674b66833fadffd8ad476。

动机 / Motivation

没有发现阻塞问题。拥有失败研究 worker 的操作者:最新文件可能在清理时丢失,而账户配额修复后原会话仍被外层调度器停住。 旧路径在失败后移除最新现场,或继续把已修复的硬配额失败挂起;本头显式保留停止的源容器,并在新获准的有限 phase 中继续原会话。 本头的真实 SQLite、worker 子进程和本地确定性 native 协议流程保留原会话、失败记录与 attempt 计数,成功继续下一项工作;Stop 和重复请求反例均通过。 本 PR 不恢复缺失历史、不把失败尾部认证为完整 checkpoint,不授予账户、预算、执行或合并权限,也不认证实际研究分数、模型健康或长期收益。 真实历史现场的正向恢复、实际安装采用、真实模型与完整科学成本/效益仍由原运行及 native qualification owner 单独验收。

改动思路 / Approach

失败现场保留属于 Docker/native provider IO,重试资格复用共享 TypeScript Host 规则,有限 phase 和原始失败的持久观察仍由现有调度器维护;这三个边界相互需要且没有新增第二个通用调度或权限 owner。 本 PR 交付显式失败保留与同会话有限接续的可审组合;普通清理、自动硬配额重试、原始记录与当前 Goal/Stop/会话准入保持原语义。 最小替代方案逐一核对:保留现状会丢失最新文件或让修复后的工作继续停住;自动重试硬配额、清空failed状态或回退旧checkpoint都会破坏原失败、历史或预算。这里沿原provider和scheduler扩展,用共享TS规则移除Journal内重复分支,不新建通用恢复框架。source digest记录本次明确意图的对象,不能代替运行、账户、资源和当前Goal授权。

触发 可观察行为 保留的边界
显式保留的solver失败或结果不确定 在归档/评分drain前停住原容器;保存或保存失败都保留源 原始退出、失败及机会/冷却不改;导出仍未取得恢复资格
原硬配额失败修复后显式请求 fresh allowance及原会话/进程/源码核验后建立有限phase 原失败、attempt、截止时间及语义replan继续保留
同一请求重复,或phase再次失败 返回同一phase及其新失败 不清旧记录、不补充时间;再次修复要绑定新的失败

具体改动 / Changes

规范采用更改前的 benchmark/README.md(spec_revision 1a309ccb3278b68f79d674b66833fadffd8ad476),并对照同版 账户耗尽操作合同 与 Governed Turn journal/retry。original_session、finite_phase、current_admission、semantic_replan、quota_repair 均由生产入口和反例验证;research_qualification 保持原有独立后续验收。没有用本PR新增说明替换原合同。

完整21路径、+1139/-52已读:15个生产路径、4个测试路径和2份文档。全部21路径分别与组件 6097e14220e03d8ceb2b1df4555e59e62a64883c 或 d442384c8259c56bbe63709b0b76106db07b5e04 字节相同;仍按本组合head核验共享调用和实际准入,没有把组件结论当作整体验收。保留源、明确修复意图、derived phase provenance和attempt回退各有实际生产触发与消费;重复请求不能用旧注释或旧digest改变现有失败。

关键代码讲解

  • FailurePreservation.finish 在任何可失败IO前标记保留,关闭observer、尝试有界完整捕获,并在provider后续归档/评分drain前停止同一容器。capture、barrier、磁盘或stop失败都不会强删唯一源;receipt只记录有限状态,导出始终 restore_qualified=false。
  • read_failed_host 逐一核对原始state digest、唯一returned wake、已退出进程组、当前Goal/Agent会话和profile、稳定的结构化quota-failed native尾部,再只读现有provider的fresh allowance。未知、prose-only、错误会话/容器或不足额度保持hold。
  • resume_state_file 在既有独占锁下创建不可变的有限phase;原文件保持。后继worker从原失败次数继续增加;旧digest的重复请求返回已存在phase,保留后来失败;下一次明确修复需要新失败身份。未完成semantic replan仍交原owner结算。
  • hostFailureRetryCheck 接过Journal已有的显式重试规则;新benchmark adapter只允许quota repair,共享Journal保留其原显式策略。新增 host_failure.retry 仅返回资格,不授予session、预算或launch权限。
  • native_failed_turn 两次读取完整native历史,验证failed尾部、结构化quota错误和原进程窗口,没有把failed改成completed;原 _complete_history 的completed checkpoint门槛继续生效。

对主干的风险 / Risks

最强反例是“保存receipt成功但评分drain时计算仍在运行”,或“重复旧修复清掉新失败/补充时间”。现有production run_agent的原缺陷oracle在早期preservation版本会因Running=True失败,修正组件已通过真实Docker/native HTTP/SQLite完整29例;本次经7个保留组件源/测试字节、依赖声明与冻结SDKpin一致性检查复用该证据,未声称重新运行Docker。对应缺失参与者、重复finish、隐私和未知stop拒绝证据保持。既有未合格环境尝试继续与有效证据分列,未改写成产品通过或性能收益。

本组合head新执行结果分开记录:三个组合模块64通过、9跳过(7项缺可选SDK、2项需隔离Docker);共享Host/外部scheduler/replan核心134通过;TS Journal/实际handler60通过。真实SQLite与生产worker subprocess覆盖原session接续、原failure/attempt保留、再次失败、Stop准备前/后、profile/session/digest/unknown或耗尽allowance拒绝,以及未决semantic replan不被清除。native账户协议由本地确定性进程提供,核验仅account/read与account/rateLimits/read,未读取真实账户、调用模型或改活动run。

另外在不可变base和本head经真实Effect turn_journal.inspect 比较9组完整恢复/错误/副作用投影,零归一化且完全相同:普通quota仍不自动重试;显式修复、capacity剩余/耗尽、output budget拒绝、畸形counter/extra字段及terminal零effect均保持。默认关闭的provider无/有普通controller的2+2实际基线对照沿相同冻结组件复用,普通退出、单次评分/提交、archive及remove结果保持;新method/SDK安装和allowance可用性不会启用保存或派发。

语义与 CI 对齐

这里扩展既有Effect方法词汇 host_failure.retry,复用failure kinds与原typed owner;benchmark专用要求留在provider侧,没有加入通用quota/Todo/settlement义务。quick premerge的6个direct与4个selected检查通过,零检查失败/警告,整体仍为benchmark-sensitive的 manual_review_required,exit1不表示合并门禁通过。远端CI按wait_for_ci=false未查询。

我的整体评价 / Verdict

APPROVE,这个组合构成有用且可逆的明确增量:失败现场能留下,修复后原会话能在独立获准的有限phase中继续;完整研究接续和科学收益继续按原owner验收。本头没有修改渲染页面;检查了 benchmark CLI、scheduler 参数、原有操作文档与失败读回,属于已有研究适配器的显式 CLI/Host 路径。 未验证范围:安装包前端、真实账户修复和实际历史现场恢复未由本次本地组合测试认证。 失败现场保留属于 Docker/native provider IO,重试资格复用共享 TypeScript Host 规则,有限 phase 和原始失败的持久观察仍由现有调度器维护;这三个边界相互需要且没有新增第二个通用调度或权限 owner。 本 PR 交付显式失败保留与同会话有限接续的可审组合;普通清理、自动硬配额重试、原始记录与当前 Goal/Stop/会话准入保持原语义。 共享TS重试规则的提取已消除一处原inline决策;provider IO和旧stopped-source兼容有实际消费者,继续保留比另造统一executor更合适。没有要求无关TS重写或新的跟踪Task。live deployment、旧历史正向恢复、真实模型及全成本收益仍未认证;评审结论、merge readiness与maintainer合并权限分别保留。

English verdict: APPROVE — HEAD b0dd6bd; exact combination, same-session finite recovery, current Stop/identity holds and shared-rule parity verified, with unchanged native component evidence explicitly reused. Live/scientific qualification and merge authority remain separate.

@loopx-agent loopx-agent closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants